1. Field of the Description
The present description relates, in general, to video game development and communications between development tools and video game platforms, and, more particularly, to systems and methods for remotely gathering performing metrics, such as frames per second, from two or more game platforms running a video game (e.g., a number of platforms or consoles or a “testing farm” concurrently running a game during performance or load testing). The metric gathering is “remote” in the sense that a majority (if not nearly all) the metrics gathering software (e.g., a performance metrics framework) is run on a computer or machine separate from the game hardware or console, and the retrieved data is stored external to each of the game platforms for further processing.
2. Relevant Background
The video game market has moved from a smaller niche market to a multi-billion dollar market. The demand for new video games is growing rapidly as the size and demographics of game players continues to expand. Development is complicated because video games used to be made for only one or two game platforms, but now there is a demand for video games that can be sold on numerous platforms including standalone game consoles, portable and desktop computers, handheld portable gaming devices, and other electronic devices such as cellphones, digital music players, and tablet computers (collectively termed “game platforms”). As a result, there is significant competition among game developers to create video games to meet the growing demand in a timely and efficient manner and, often, there is a requirement that each of these new games be able to run as desired (e.g., similarly) on differing game platforms, devices, or consoles.
Large-scale, commercial video games are typically created by large development teams with the development process taking one to three years and costing millions of dollars. A development team may include producers to oversee production, game designers, artists, programmers, level designers, sound engineers, and testers with some team members handling more than one role. The development team works in a collaborative manner to create game content (or game assets) and game code, which together may be thought of as a game application, which can be run by a gaming engine on a particular game platform to provide the game to a player.
For example, programmers may write new source code, artists may develop game assets such as characters or scene objects, coloring schemes, textures, 3D models of game elements, and the like, sound engineers may develop sound effects and music, writers may create character dialog, and level designers may create advanced and eye-catching levels. To support these team members, numerous game development tools, such as Microsoft XNA, Maya from Autodesk, Inc., and the like, are now available for use in designing and creating game content including creating and modeling characters and even for defining game logic (e.g., how high a character jumps, how much life does a character gain or lose based on a game event, how fast does a character or game element move, and so on). Additionally, each video game console or platform developer typically will make a software development kit (or SDK or devkit) available to game developers, and each SDK may include applications or tools that are useful in creating new video games. Each SDK may also include a communications library and/or interface (e.g., platform communications data) to facilitate communications with a game platform (e.g., the platform running a game under current development or running a built game for testing or use by a developer to see how a new character, asset, game parameter, or the like is working in the game).
Another ongoing challenge for video game developers is how to best complete performance or load testing. During game development, there is typically a need to gather performance metrics from the game platform or console, such as current frames per second (FPS). These performance metrics are used by game developers to determine or verify how well the game application is running on a particular platform such as in terms of frame rate (e.g., FPS), memory (e.g., megabytes used and remaining on the platform), and even how these and other resources are being used by the various subsystems of the game platform including the physics simulation subsystem, the rendering subsystem, the artificial intelligence subsystems, and the like.
Presently, a custom solution is designed for each game platform, and the custom solution or on-platform metrics gatherer acts to determine which metrics to track and gather and how to transmit this information from the game platform to a developer's computer system. Such custom solutions have presented a number of problems. The custom metric gathering tools are time consuming and expensive to develop and maintain. Additionally, writing a custom software communications solution on a per-platform basis led to divergence between both the way the information was transmitted and the format of the reported metrics, which made aggregation and analysis of metrics data from varying game platforms difficult and more complex.
In some existing performance testing systems, the on-platform gathering tool writes a fixed (or predefined) set of performance metrics to a file on the game platform or console. This is problematic since file systems differ significantly between differing types of game platforms, which makes aggregation and analysis of the data more complex. Further, some game platforms may not have enough disk space to store all the metrics that a developer may want to gather from the game. In addition, some game platforms do not have a built-in hard drive so that the metrics cannot be stored to such a file. Storing the metrics on the game platform also requires the data to later be mined from a text file, and such data mining can be time consuming and vary with each platform type. Further, the data mining requires human supervision that slows down the feedback loop of information back to the game developers.
Significantly, it is also undesirable to have the performance metrics gathering application to be run on the same machine or platform that is running the video game being performance or load tested. Such operations can use memory and processing time such that proper testing practices are violated as the testing environment is polluted or changed, which can create inaccurate test results. Specifically, a video game may be created that functions as desired on a platform until the platform hardware is also burdened with performance metric gathering and data storage, which may result in the performance metrics differing from those occurring when performance testing software and files are removed from the platform (e.g., the performance/load test itself may impact the results of the test).
Hence, there remains a need for improved methods and systems for gathering, aggregating, and analyzing performance metrics from various types of game platforms running game applications or software that is under development. Preferably, such methods and systems would facilitate testing games created for use on a variety of differing video game platforms (e.g., performance or load test of a game on two or more differing game consoles). Further, it is preferred that the metrics gathering methods and systems be adapted to better comply with testing protocols than prior performance/load testing by not significantly modifying the test environment (e.g., not polluting the test by altering operation of the platform hardware or the game itself).